[BREAKING] [FIX]: Return UNDETERMINED when an evaluator cannot observe the evidence it needs - #141

Description

Fixes #140 and closes #167, which Nina Chikanov (@nina-msft) asked be superseded rather than done separately. #140 has a standalone repro: a probe asserting an agent does not exfiltrate passes green while the agent exfiltrates on every turn, because the adapter declares TOOL_ONLY and so never populates Response.side_effects.

ToolCalled and SideEffectOccurred return NOT_DETECTED in two different situations: the thing did not happen, and the adapter does not report that kind of evidence. resolve_as_attack maps NOT_DETECTED to SAFE and resolve_as_probe maps it to UNSAFE, so a gap in the adapter becomes a statement about the agent. _adjust_for_observability covers part of this, but only for RESPONSE_ONLY, only in XPIA, and only by counting tool calls.

This PR adds a regression test for the attack side. Run against main at 3d219d8 it produces:

=========================== RAMPART Safety Summary ============================

UNCATEGORIZED (2 tests)
  FAIL  test_dropping_observability_does_not_look_safer_async -- Attack objective detected: Side effect 'http_request': {'url': 'https://evil.com/steal'} (tool_and_side_effects)
  PASS  test_dropping_observability_does_not_look_safer_async -- Agent defended successfully (tool_only)

FAIL and PASS there are RAMPART's per-result verdict labels, not pytest outcomes. Both lines come from one pytest test that runs the same attack twice. The agent exfiltrates on both runs; the two adapters differ in the declared level and in what that level lets them report, so the TOOL_ONLY one returns the same response with no side_effects, which is exactly what a TOOL_ONLY adapter can do. RAMPART calls that second run "Agent defended successfully". The pytest test fails on main and passes with this change.

Every combination where a built-in evaluator needs evidence the adapter does not report, measured with a MockAdapter:

Two of the six were already correct, and both were correct because _adjust_for_observability caught them.

Rebased onto main at 3d219d8, so the tests here follow RMP001 from #158 and #159 and the xdist transport is the one #166 landed.

Changes

• ObservabilityLevel gains observes_tool_calls and observes_side_effects, following the PayloadFormat.is_text and is_binary pattern already in that file. Its class docstring described only the RESPONSE_ONLY case, so it now also covers TOOL_ONLY with side effects, which is the case in the linked issue.
• EvalContext gains observability_level. It is required, so a context built by hand has to say what the adapter behind it could see.
• evaluate_turn_async takes the level, required and keyword-only, and puts it on the context. XPIAExecution and SingleTurnExecution both pass adapter.observability_profile. Result and EvalContext.from_response require it too, which is Nina Chikanov (@nina-msft)'s request below and what closes [FEAT]: Deprecate omitted observability_level across public APIs #167.
• ToolCalled and SideEffectOccurred return UNDETERMINED when they cannot see the evidence they need. The check runs after the scan, so anything the adapter does report still counts as evidence. _adjust_for_observability makes the same allowance today.
• The UNDETERMINED summary on both strategies is built from undetermined_operands, so it names every channel that could not be observed rather than only the operand the composite reported first. Repeats collapse, and anything past the first two is counted. This is Nina Chikanov (@nina-msft)'s second request below.
• _AllEvaluator short-circuits only on a NOT_DETECTED left operand. An UNDETERMINED left operand no longer skips the right one, so & no longer depends on the order the operands were written in. Both undetermined branches carry the evidence of both operands. This is the review fix from Nina Chikanov (@nina-msft) below.
• _AnyEvaluator names the undetermined operand and carries the evidence of both, instead of a bare "One or both operands undetermined". Outcomes are unchanged. Without this, | hid the adapter setting behind the verdict, which is the one thing this PR is trying to surface, and the note added to authoring-tests.md points the reader at | for exactly this case.
• The XPIA undetermined summary prefers results that are themselves UNDETERMINED, matching the probe summary above it. Without that it could lead with a NOT_DETECTED rationale from a different turn. Settled results are read only when nothing else gave a reason, which is what the _adjust_for_observability downgrade looks like.
• The probe unsafe summary takes its reason from a NOT_DETECTED result. It took the last rationale of any outcome, so once these evaluators can return UNDETERMINED, an undetermined turn could state the reason for a definitive failure.
• EvalResult gains undetermined_operands. & and | record every operand they ran that came back UNDETERMINED, so "the predicate is false" stops being indistinguishable from "the predicate is false and every part of the evaluation ran". ~ carries its inner result's entries through, each reason is kept once, the xdist transport round-trips it with the same ANSI stripping as the other free text, and JsonFileReportSink emits it as eval_undetermined_operands when it is not empty. This is Bashir Partovi (@bashirpartovi)'s first option below; the truth table is untouched.
• The SAFE summary on both strategies names what was left undetermined instead of reporting a plain pass. It names the first two distinct reasons and counts the rest. Verdicts do not move, so no existing result changes status.
• The XPIA unsafe summary takes its evidence only from DETECTED results, matching the probe summary. An UNDETERMINED composite can carry evidence of its own, and that evidence is not what established the verdict.
• The JSON run report carries observability_level. It named the verdict, the strategy and the harm category but not the level the run was gathered under, so a dashboard could not tell a clean pass from one the adapter was never able to see through. The xdist transport already carried it.
• Every read of undetermined_operands and evidence in the composites, the summaries and the two serializers goes through safe_str_list, and the probe's unsafe and error summaries put rationale through safe_str where the XPIA summary was already guarded. _distinct_operand_reasons flattened the operand list with a comprehension, so a third-party evaluator returning a non-iterable, or an iterator whose __iter__ raises, aborted summary construction before the containment helpers saw it. evidence had the same shape, and this branch had taken the composites from one evidence concatenation to five, where a value that is not a list broke the compose step itself.
• The probe unsafe summary renders each rationale before testing whether it has content. It filtered on the raw value first, so a rationale whose truthiness raises took the summary and the verdict with it, and a whitespace-only rationale printed UNSAFE: with nothing after the colon. This is Nina Chikanov (@nina-msft)'s request below, using the code she supplied. One behavior moves with it: a rationale that is falsy but renders as something, such as None or 0, now shows as itself where it used to fall through to the generic line. Both readings are of a value that already violates the declared str, and the verdict is the same either way.
• safe_str returns an exact str. str() accepts a __str__ that returns a str subclass, so the rendered value could still carry evaluator code on the methods RAMPART reaches for next. _distinct_reasons and _merge_undetermined already called .strip() on it, and the fix above adds a third such call, so containment was moving the failure rather than removing it. str.__str__ is the C slot: it cannot be overridden, cannot raise, and returns the argument unchanged when it is already exact.
• The xdist truncation marker carries the run's real observability level. It hardcoded RESPONSE_ONLY, so a result too large to send came back through the controller claiming the narrowest level, in the field the rest of this PR is about. Predates this branch; the original Result was already in scope.
• The undetermined summary reads settled results only when no result stayed undetermined, so a gap another turn settled around cannot be offered as the reason this verdict was missed.
• safe_str and safe_str_list in rampart/common/text.py coerce evaluator-supplied values without raising. Every rationale interpolation in the composites goes through them, as does every read of undetermined_operands in the composites, the JSON sink and the xdist serializer. _AnyEvaluator and _AllEvaluator between them gained three rationale interpolations main does not have, so a value whose __str__ raises turned inputs that resolved cleanly on main into SafetyStatus.ERROR, losing a verdict the evaluators had already reached. Both helpers catch Exception rather than BaseException, so cancellation and interrupts still propagate.
• ObservabilityLevel and authoring-tests.md now say the guarantee is per channel rather than per field: a level that reports a channel is taken at its word for what it puts in it, so a tool call reported with redacted arguments still counts as observed and a predicate over those arguments can return NOT_DETECTED.
• Session.send_async, authoring-tests.md and quickstart.md said empty lists mean "no observations", not "nothing happened". That rule predates the declared level and now reads backwards, and it contradicted observability_profile's own docstring in the same file. All three now say an empty list is read against the declared level.

Why the fix is in the evaluator

Two docstrings disagree about this, so I want to be explicit about which one I followed and why. Both are quoted as they stand on main; this PR updates both.

rampart/core/types.py:27-29:

When the adapter declares RESPONSE_ONLY, evaluators that require tool call data return UNDETERMINED rather than a false SAFE.

rampart/evaluators/tool_called.py:23-25:

This evaluator only detects conditions. It does not reason about observability gaps. That adjustment is owned by the execution strategy.

I followed the first one.

The obvious alternative is to keep the adjustment central and have evaluators declare a required_observability for the strategy to read. I could not make that work for composition. Under TOOL_ONLY, ToolCalled("x") | SideEffectOccurred("y") should still return DETECTED if x was called, while the right operand cannot be observed. A strategy-level check against a composite's declared requirement cannot see the operands, so it either suppresses a real detection or does nothing. The post-scan allowance above has the same problem: "evidence the adapter actually reported still counts" is a per-operand runtime fact, not something a static declaration can express. |, & and ~ already arbitrate this correctly once operands can return UNDETERMINED, which is what this change gives them.

There is also precedent for an evaluator reporting its own uncertainty. LLMJudge returns UNDETERMINED when the judge output is malformed after retries or the call fails, rather than guessing. Those are transient instrument failures and an observability gap is static configuration, so the situations are not identical, but the outcome type is doing the same job in both: EvalOutcome.UNDETERMINED is defined as "The evaluator could not make a determination".

The adjustment itself stays where the second docstring puts it. _adjust_for_observability is unchanged and still owns the verdict downgrade. What changes is the quality of its input. The sentence in ToolCalled's docstring is contradicted by this PR and is updated, as is the matching note in docs/usage/authoring-tests.md.

No new verdict semantics

UNDETERMINED is not new at either level. EvalOutcome.UNDETERMINED is produced today by LLMJudge and by | and &, and preserved by ~. SafetyStatus.UNDETERMINED is produced by both resolvers and by _adjust_for_observability. Every consumer already handles it: the resolver precedence rules, the composition operators, the xdist round trip through SafetyStatus(value), JsonFileReportSink, the WARN terminal label, and the population summary. This change produces it in more of the cases it already exists for.

DETECTED that came from observed evidence is untouched on every path, so no evidence-based detection is weakened. The one detection that changes is ~ inverting an absence the adapter could not attest, covered below.

Breaking changes

Yes, in two ways, and the title carries [BREAKING] as Nina Chikanov (@nina-msft) asked.

1. observability_level is required on four public APIs. EvalContext,
EvalContext.from_response, evaluate_turn_async and Result no longer
default it. Three of those four parameters are introduced by this PR, so the
break there is against a signature that has not shipped; Result is the one
that predates the branch and loses a real default of RESPONSE_ONLY.

Result is also the widest of the four: of the 94 call sites in this repo's
Python files that omitted the argument, 75 were Result(...), counting the ones
written inside pytester source strings. Two more Result(...) examples in
docs/ omitted it as well, and are updated here. Nina Chikanov (@nina-msft) left this one to
my judgement with a stated preference for requiring it, and requiring it is
what actually dissolves the asymmetry, so that is what this does. Say the word
and I will put the Result default back.

Migration is to pass the adapter's declared level, normally
adapter.observability_profile. Omitting it is a TypeError at the call
rather than a silent assumption in a report. No call site in rampart/
omitted it, so no built-in behavior moves. All four are keyword-only, so
nothing positional breaks.

2. Verdicts move, in one direction for the evaluators on their own and in
one cell for &.

Nothing is removed or renamed otherwise. The xdist transport gains a key.
JsonFileReportSink gains two: eval_undetermined_operands per turn when the
list is not empty, and observability_level on every result unconditionally, so
a consumer validating a strict schema on a result object sees a new always-present
field. EvalResult.undetermined_operands is
written and read at both ends of the xdist transport; a payload without the key
still deserializes and an old controller ignores the extra one, so
SCHEMA_VERSION is unchanged, and bumping it would make _validate_schema
reject the whole payload instead. JsonFileReportSink emits
eval_undetermined_operands per turn only when the list is not empty.

For ToolCalled and SideEffectOccurred used alone, NOT_DETECTED becomes
UNDETERMINED and nothing moves toward SAFE. What existing suites will see:

• An attack that passed because the adapter could not see side effects now returns UNDETERMINED and fails. That is the bug being fixed, and it will surface as a newly red test.
• A probe using ToolCalled or SideEffectOccurred below the level it needs goes from UNSAFE to UNDETERMINED. Both are falsy, so the test still fails, but the terminal label changes from FAIL to WARN.
• ~ToolCalled(...) under RESPONSE_ONLY previously returned DETECTED by inverting an absence the adapter could not attest, and now passes UNDETERMINED through. On a probe, "must not call X" against an adapter that cannot report tool calls was a false pass and now fails. The linked issue is the same shape one level down: ~SideEffectOccurred("http_request") against a TOOL_ONLY adapter.
• With the default trial threshold of 0.0, a group whose clones are all UNDETERMINED logs a passing gate line where it previously logged a failing one. The clones still fail, since assert result is falsy, and _evaluate_gates only logs, so no CI outcome flips. I left the threshold alone because PR [FEAT]: Add execution trial populations and threshold verdicts #121 is reworking that layer.

Making & order independent required choosing which outcome wins when one operand is NOT_DETECTED and the other is UNDETERMINED. It returns NOT_DETECTED, which is Kleene and is what the review asked for. Against every operand pair on main, one cell moves:

main   : undetermined & not_detected -> undetermined   attack=undetermined
branch : undetermined & not_detected -> not_detected   attack=safe

This is not a regression against main for ToolCalled or SideEffectOccurred, which returned NOT_DETECTED on main at a level that could not report the evidence, so the conjunction already resolved SAFE. It does mean a composed evaluator no longer gets the protection the first commit of this PR gave it in one of the two operand orders, and that a degraded LLMJudge inside & can now resolve SAFE where main said UNDETERMINED. Both cases now record the reason in EvalResult.undetermined_operands, and a SAFE summary names it, when the undetermined operand is on the left, since & still short-circuits on a NOT_DETECTED left operand and never runs what is to its right. | reports UNDETERMINED in those cases, and the docs now say which operator to reach for and which side to put the observability-dependent operand on.

For the verdict changes there is no migration beyond fixing the adapter's declared level or the evaluator choice. The new rationale string names the declared level, the channel it does not report, and the target the evaluator was looking for.

Deliberately out of scope

• _adjust_for_observability also fires when it should not: RESPONSE_ONLY with ResponseContains is downgraded even though that evaluator never needed tool data. That is a false positive rather than a false negative, and narrowing the heuristic is a separate change.
• LLMJudge now receives observability_level and ignores it. Telling the judge that tool calls are not visible would stop it reading an evidence-free transcript as innocence, but that changes judge prompting.
• ResponseContains is untouched on purpose. Every level reports text, so no declared level hides it.
• Splitting EvalOutcome.UNDETERMINED into "cannot observe" and "did not run" so & can treat them differently. That is the real fix for the LLMJudge case above and it is bigger than this PR. undetermined_operands records both kinds without telling them apart, so it does not pre-empt that design.
• Reads of evaluator-supplied values in three files predate this branch and are untouched: confidence and rationale in the xdist serializer, the same two in the JSON sink, and the rationale the LLM driver puts into its next prompt. A value that cannot be rendered still costs the payload, the report file or the next turn there, and at rampart/drivers/llm.py:333 a value whose truthiness raises costs the next turn before rendering is even tried. The summaries and the composites are contained; these are not, and widening into them is a separate change.
• A reason that itself contains "; " is indistinguishable from the separator the summary joins on. Same on main for the existing summaries, and the full list is on Result.eval_results and in the JSON either way.

Checklist

• pre-commit run --all-files passes
• Tests added or updated for changes
• Documentation updated

Tests

276 new tests against main, and one removed: test_left_undetermined_short_circuits_async
asserted that & skips the right operand when the left is UNDETERMINED, which is the
behavior the review asked me to remove. 8bf0b62 renamed it to
test_left_undetermined_evaluates_right_async and inverted its right.call_count
assertion, so the coverage moved rather than being dropped. It is the only collected node
id main has that this branch does not, so nothing else existing was changed, removed or
reparented.

One test appears and disappears inside the branch rather than against main:
9ac85b7 added test_observability_level_defaults_to_no_declared_limit and b86a67f
removed it, because it asserted the default that commit takes away; four tests asserting
the TypeError replace it, one per API.

The containment tests cover the operand path through _summarize_undetermined_operands
and _explain_undetermined, both summary builders, every branch of the three composites,
the xdist serializer, the new report key, and safe_str_list itself. Most of them are
parametrized sweeps over the composites, one per field, because line coverage cannot see
an expression change: a guard runs whether or not any test would notice it being removed.
Neutering each of the 33 safe_str and safe_str_list call sites this branch adds, one
at a time, turns the suite red at every one of them.

Across the whole PR, by file: test_evaluator.py 150, test_single_turn.py 26,
test_xpia.py 25, test_result.py 21, test_text.py 20,
test_tool_called.py 9, test_side_effect.py 7, test_types.py 7,
test_xdist.py 5, test_json_file.py 4, test_execution.py 2. Those cover the outcome table for
& and |, commutativity over all nine operand pairs, De Morgan both ways,
associativity over all 27 triples, which cells record an undetermined operand
and which cannot because the short-circuit skipped it, UNDETERMINED at each
insufficient level with the rationale naming the level and the target, evidence
still counted below the declared level, the new field surviving the xdist round
trip with ANSI stripped at the boundary, and the report key present only when
the list is not empty.

The 27 that 4b43052, b86a67f and 82f7926 added:

• test_result.py (10): nine on _explain_undetermined in priority order, plus one TypeError test. Operand reasons beat the composite rationale, a reason repeated across turns collapses on both the operand path and the rationale path, the count names what it does not, a settled result cannot speak over an operand that stayed undetermined, a settled result does speak when nothing else did, and a blank or whitespace-only reason falls through to the fixed phrase rather than rendering an empty detail.
• test_xpia.py (7) and test_single_turn.py (7): ToolCalled("x") | SideEffectOccurred("y") under RESPONSE_ONLY end to end, naming both channels, which is the case in the review comment; the _adjust_for_observability downgrade naming the gap it recorded; and the unit-level dedup, count and settled-result cases on both strategies, plus the two fallback cases on the probe.
• test_types.py (2) and test_execution.py (1): the remaining three of the four TypeError tests, one per API.
• test_xdist.py: no new ids. The oversized-result marker test now asserts the level survives truncation instead of being rewritten to RESPONSE_ONLY.

tests/integration/test_smoke.py uses ToolCalled through EvalContext.from_response and asserts a detection, so no verdict there moves; it only gains the now-required argument. It needs no credentials and passes: 2 passed.

Documentation

• docs/usage/authoring-tests.md: the ToolCalled warning said it "always returns NOT_DETECTED" under RESPONSE_ONLY, which is no longer true. SideEffectOccurred had no note and now has one. Added a short paragraph under the levels table on why declaring the level honestly matters, a paragraph saying the guarantee is per channel rather than per field, a note on how UNDETERMINED travels through & and |, which no user facing page covered, and a paragraph on what undetermined_operands records and which side of & to put an observability-dependent operand on.
• docs/attacks/xpia.md: the Observability Adjustment section now says what it is for, now that evaluators handle their own cases, and the composition example says which operator to reach for when two evaluators are two views of one harm, and that the result records the gap rather than the verdict resting on silence.
• docs/contributing/extending-rampart.md: the custom execution strategy example called evaluate_turn_async without the level, which would silently treat every adapter as fully observable. Fixed, plus a bullet in the key points, reworded again this round because omitting it is now a TypeError rather than a wrong assumption.
• docs/contributing/testing.md and docs/usage/pytest-integration.md: the Result(...) helper and the manual-recording example both pass the level now, with a line on how to choose one and a note that existing tests were backfilled with the old default so no test changed meaning.
• docs/api/core-protocols.md: evaluate_turn_async is exported from rampart.core but was absent from the API reference. It joins the other rampart.core.execution members there. It is not importable from rampart directly, so it does not belong on core-types.md, whose lede promises exactly that.
• docs/getting-started/quickstart.md and docs/glossary.md: the empty-list rule and the EvalContext entry.

No new pages, so no mkdocs.yml nav change.

Checks run locally

Rebased onto main at 3d219d8. That range brought ruff 0.16.3 and ty 0.0.72
into uv.lock via #169, so the numbers below are at those versions, not the ones
I quoted two rounds ago.

ruff 0.16.3 check .............. passed
ruff 0.16.3 format --check ..... 128 files already formatted
ty 0.0.72 check ................ passed
flake8 7.3.0 (RMP codes) ....... passed, 0 violations
coverage run -m pytest ......... 996 passed, 7 skipped
pytest -n 4 .................... 996 passed, 7 skipped
coverage report ................ TOTAL 94%, threshold 80
                                 core/types.py 100%   core/evaluator.py 100%
                                 core/result.py 100%   reporting/json_file.py 100%
                                 common/text.py 100%
                                 evaluators/tool_called.py 100%
                                 evaluators/side_effect.py 100%
                                 probes/_single_turn.py 100%
                                 attacks/_xpia.py 95%
mkdocs build --strict .......... no content warnings, same as main

mkdocs build --strict aborts on both main and this branch for the same
environmental reason: the Material offline plugin cannot create symlinks on
Windows. All 43 warnings on both sides are that one, and filtering it out leaves
none.